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ABSTRACT 

This paper provides advice to college and university 
administrators on issues and problems associated with the assessment 
and documentation of institutional effectiveness, in light of the 
current Southern Association of Colleges and Schools (SACS) criteria 
for institutional self-evaluation. Administrators need to: (1) get 

the attention of high-level decision-makers at their institution; (2) 
not just throw money at the problem; (3) avoid setting themselves up 
for failure by using poor definitions; (A) avoid assigning the 
project to their institutional research office; (5) avoid stopping 
the process once they get the results; (6) avoid getting lost in 
small numbers; (7) avoid a total focus on tests or numbers; (8) not 
try to hide issues; and (9) avoid using the SACS self study as a 
promotional document for the institution. (MDM) 
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Issues/Problems in Assessing and Documenting Institutional Effectiveness 

John Losak 
Nova University 
January 1994* 

Virtually all so-called "problems" are relational in nature; that is, the intensity of the 
problem is only meaningful with respect to the context in which the problem occurs. To speak 
about "problems" or issues with respect to assessing institutional effectiveness, it may be helpful 
for a moment to describe the contexts from which institutional effectiveness measures are 
emerging. 

Historically, institutions have been measured by outside agencies, lay citizens, and their 
respective legislators if they are public institutions or by benefactors if they are private 
institutions, on a variety of criteria, some of which are vague and personal; others of which are 
more specific and public. With respect to formal evaluations from discipline areas, i.e., the 
National League for Nursing, the American Medical Association, or the American Psychological 
Association, departments and divisions within a college or university have usually been held to 
a fairly systematic and routine set of criteria which have emerged within the discipline over time. 
With respect to traditional (process) institutional effectiveness measures as developed by 
accrediting agencies such as the Southern Association of Colleges and Schools (SACS), these for 
the most part were relatively easy to accumulate and document. 

The presentation of data for a SACS reaccreditation study prior to 1985 amounted to an 
enormous effort once a decade to gather together already available information, e.g., number of 
faculty who have master’s and doctoral degrees, the ratio of these faculty to students, the number 
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of library books, documentation of fiscal responsibility, and other like information. The 
important difference between the collection of traditional data describing institutional resources 
and the current expectations of the SACS Criteria are: (1) that the new information being sought 
is not something that can be gathered only once a decade, (2) the responsibility is left to the 
collective professional judgments of college personnel — not SACS, (3) the Criteria places a 
heavy emphasis on documenting institutional effectiveness. The institution is expected to devote 
energy and resources to regularly gather information which in all probability was not routinely, 
systematically, and publicly gathered by the institution prior to the initiation of the new SACS 
statements. It may be that some institutions collect information reflective of student outcomes, 
but there are many colleges and universities which have this information available only 
sporadically and then rarely in relation to the stated goals of the university or college. Thus, the 
major difference and the essential core of issues from which problems may arise is the 
requirement that something additional be undertaken by the college or university to meet the 
expectations of the Southern Association. This "something" is expounded in the Resource 
Manual that was developed to accompany the SACS Criteria publication. The Resource Manual 
basically attempts to elaborate some of the methodologies by which issues might be addressed 
with regard to assessing institutional effectiveness. 

However, there is little focus in either the Criteria or the Resource Manual on 
concomitant problems associated with implementing the programs necessary to accomplish an 
assessment of institutional effectiveness as envisioned by SACS. With these brief contextual 
comments, I will now move to address specific issues/problems which can be anticipated in the 
assessment of institutional effectiveness based on outcomes measures. 
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Get the attention of high-level decision-makers at your college or university. 

This can be a severe impediment if not accomplished. Although many of us who 
have been directly involved with the SACS Criteria for several years believe that the 
issues are clear-cut and that colleges and universities should be moving with all deliberate 
speed to implement institutional effectiveness programs, I think that there are a reasonably 
large number of institutions where the chief administrative officers have yet to address 
seriously the new expectations for assessment of institutional effectiveness by which their 
reaccreditation will be evaluated. This view is strengthened by the large number of 
recommendations given by Visiting Teams in the general area of institutional 
effectiveness. 

Don’t just throw $$ at the problem. 

This second problem relates to resource allocation. I believe there needs to be a 
fine balancing act between an expenditure of resources which would so heavily burden 
the institution that it would be impractical to implement, and, on the other hand, a total 
neglect of resource needs. As a fairly mundane example, let me point out that the follow- 
up of graduates into employment as a measure of institutional effectiveness related to 
student outcomes need not be an every year event with every student. To burden the 
institution with such a heavy demand would require more outlay of resources than is 
needed to answer the question. Rather, a sample of students followed up every second 
or third year would probably answer the question sufficiently well. Although this is a 
minor example, I think the general notion of trying to sample and to minimize 
expenditures of resources and efforts is extremely important. 
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Don’t set yourself up for failure by using poor definitions. 



From a conceptual level, this may be the most serious of all of the problems 
facing us in the area of institutional effectiveness measures. The problem that I have in 
mind is one of determining reasonableness with regard to outcomes measures. This is a 
particularly sensitive area and one which I believe gets all too little attention in the 
planning stages. 

As you begin to implement outcomes measures, it becomes almost immediately 
apparent that the game can be won or lost based on how one determines what a 
reasonable outcome measure is. Let me try to give a few examples. With regard to 
improving student retention rates, one could immediately set the institution up for success 
by saying that we hope to improve the graduation rate to 20% of the entering freshmen 
over a three-year period. For most colleges, this is a very low-level outcome measure and 
is probably already attained by most colleges. On the other hand, you could set your 
institution for immediate failure by building in a goal to achieve a 100% graduation rate 
of all first-time-in-college students. What is a reasonable goal to establish for your college 
with respect to graduation rates? In many instances, this issue is addressed by speaking 
of improvement over already existing achievement levels, and indeed one can anticipate 
moving from e.g. an historical 40% graduation rate after five years to a 45% graduation 
rate two years hence, to a 50% graduation rate five years hence. However, the same 
concept is involved with regard to improvement as it is to establishing goals initially; i.e., 
what is a reasonable rate of improvement and over what period of time should this be 
expected? 
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4. Don’t assign the project to your institutional research office. 

I believe a more subtle problem that awaits us has to do with where the task is 
assigned and carried out. If the task for assessing institutional effectiveness is assigned 
to institutional research at your college or university this could in itself be a problem, 
because then the entire project is assumed to occur outside of teaching faculty 
responsibility. In my view, institutional effectiveness measures should be directed and 
controlled by faculty and/or those persons most affected by the particular measure. That 
is to say, if the measures have to do with financial aid operations, then, obviously, the 
staff in financial aid should determine the outcome measures and the processes to achieve 
them, rather than directing this task to the institutional research office. They should as 
well be involved with the collection and interpretation of the information. The point here 
is to emphasize the need for a broad-based and rather firm commitment on the part of all 
who will be affected and not to leave this as simply another assignment with institutional 
research, because then the basic intent of the Criteria and the emphasis on institutional 
effectiveness measures will be undermined. 

5. Don’t stop when you get the results into a report 

The focus of Section HI of the Criteria is on getting back the information from 
studies conducted during the Self Study process in a manner that affects decision-making. 
It appears clear both conceptually and in terms of the requirements from the Criteria that 
the intent of self study is undermined if we do not attend to the findings in addressing 
curriculum changes and addressing methods of improving student learning as a result of 
analysis of the findings. Moreover, these findings need to be articulated to teaching 
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faculty who are the primary interactors with students. It is quite easy, given all of the 
discussions and the numbers and the problems with politics, to forget that students remain 
our primary concern, the primary focus for all of these introspective efforts. 

6. Don’t get lost in small numbers. 

Sometimes, studies conducted under the umbrella of outcomes assessment are 
communicated in a series of reports in which the basic issues get lost in tests of 
significance. By the way, that may be a good reason to keep the assignment of 
institutional effectiveness out of the institutional research office. The idea is to 
focus on broad questions which are meaningful for parents, for older students, for 
students newly out of high schools, or for your boards of trustees. For example, 
to what extent are students who arrive on the doorsteps of your college or 
university receiving an educational experience for which they signed on. Another 
broad question has to do with the following: are there any particular groups of 
students who appear to exit the university experience earlier than others? These 
are the kinds of broad issues that parents, potential students, and boards of trustees 
members want to know answers to and the kinds of questions that can be 
addressed and presented without moving into arcane uses of statistics. Case 
studies are an extremely important methodological process for getting information 
and should not be forgotten as you undertake your studies. 

7. Avoid a total focus on tests and numbers. 

Many of us continue to remind ourselves and our colleagues that qualitative 
outcomes are at least as important, if not more important, than quantitative 
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measures and we need to continue to ask questions which can be answered 
qualitatively. 

8. Do not try to hide issues. 

As a Visiting Team member for SACS, it has been my experience that few major 
issues that are troublesome can be hidden. Moreover, to do so undermines the 
purpose of the Self Study. In the written report be as forthcoming as possible 
regarding areas in which the Criteria are not met, and put forward the institution’s 
plans for corrective action. 

9. Do not use this self study as a promotional document for the university. 

The intent of the Self Study is a thorough analysis of the entire institution in an 
effort to improve the educational process for students, faculty, and staff. It is inevitable 
that in undergoing this analysis, any modem institution is going to discover areas which 
need revision and areas of functioning which need to be discarded. Criticisms of current 
management practices are often common. Each of these should be brought to light in a 
professional manner and recommendations made by the Self Study process for self- 
improvement. There will be areas of strength in which the institution may rightly take 
pride and these might be noted, but keep in mind that self-promotional activities of a 
public relations nature should take place in a forum other than the Self Study process. 

Setting forth these issues/problems and guidelines for avoiding them is an attempt to encourage 
critical examination of potential pitfalls as you move forward in the process of self-examination 
to achieve educational improvement. 
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